-
-
Notifications
You must be signed in to change notification settings - Fork 99
Drop locks from the cheroot.ssl.pyopenssl.SSLConnection method
#787
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
cheroot/ssl/pyopenssl.py
Outdated
| proxy_methods = ( | ||
| 'get_context', | ||
| 'pending', | ||
| 'send', | ||
| 'write', | ||
| 'recv', | ||
| 'read', | ||
| 'renegotiate', | ||
| 'bind', | ||
| 'listen', | ||
| 'connect', | ||
| 'accept', | ||
| 'setblocking', | ||
| 'fileno', | ||
| 'close', | ||
| 'get_cipher_list', | ||
| 'getpeername', | ||
| 'getsockname', | ||
| 'getsockopt', | ||
| 'setsockopt', | ||
| 'makefile', | ||
| 'get_app_data', | ||
| 'set_app_data', | ||
| 'state_string', | ||
| 'sock_shutdown', | ||
| 'get_peer_certificate', | ||
| 'want_read', | ||
| 'want_write', | ||
| 'set_connect_state', | ||
| 'set_accept_state', | ||
| 'connect_ex', | ||
| 'sendall', | ||
| 'settimeout', | ||
| 'gettimeout', | ||
| 'shutdown', | ||
| ) | ||
| proxy_methods_no_args = ('shutdown',) | ||
| proxy_props = ('family',) | ||
|
|
||
| @staticmethod | ||
| def _lock_decorator(method): | ||
| """Create the thread-safe, error-translating proxy method.""" | ||
|
|
||
| def proxy_wrapper(self, *args): | ||
| new_args = ( | ||
| args[:] | ||
| if method not in SSLConnection.proxy_methods_no_args | ||
| else [] | ||
| ) | ||
| with self._lock, _morph_syscall_to_connection_error(method): | ||
| return getattr(self._ssl_conn, method)(*new_args) | ||
|
|
||
| # Doesn't work via super() for some reason. | ||
| # Falling back to type() instead: | ||
| return type(name, bases, nmspc) | ||
| proxy_wrapper.__name__ = method | ||
| return proxy_wrapper | ||
|
|
||
| @staticmethod | ||
| def _make_property(property_): | ||
| """Create the property proxy.""" | ||
|
|
||
| class SSLConnection(metaclass=SSLConnectionProxyMeta): | ||
| r"""A thread-safe wrapper for an ``SSL.Connection``. | ||
| def proxy_prop_wrapper(self): | ||
| return getattr(self._ssl_conn, property_) | ||
|
|
||
| :param tuple args: the arguments to create the wrapped \ | ||
| :py:class:`SSL.Connection(*args) \ | ||
| <pyopenssl:OpenSSL.SSL.Connection>` | ||
| """ | ||
| proxy_prop_wrapper.__name__ = property_ | ||
| return property(proxy_prop_wrapper) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would expect this to be somewhere in the helper, though. So we wouldn't be adding public attributes.
Additionally, I don't fully understand the property patching bit — why is it necessary? Also, do does the connection object even need to be made thread-safe? Why?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've revamped the whole thing - getting rid of the meta class, decorator, property patching, and avoiding adding any public attributes. See what you think. Great question about thread-safety. I think you are right - I don't think it's needed as the usual pattern involves one connection per thread. If so we can further simplify the SSLConnection class. Let me know if you think we should remove.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@julianz- so I did some research and here's my observations:
- mass-searching source code, it seems like only CherryPy/Cheroot (often being vendored into something else) have any references to thread-safety: https://grep.app/search?f.lang=Python®exp=true&q=%28threadsafe|thread-safe|thread+safe%29.*\n.*\n.*SSL\.Connection (CherryPy is what first had this module until the HTTP/WSGI part was split out of it).
- mature projects like urllib3 that also make use of PyOpenSSL don't have any locking: https://github.com/urllib3/urllib3/blob/b20c836/src/urllib3/contrib/pyopenssl.py (although, it might be interesting to look at how they organize their socket wrapper)
- somebody asked about thread-safety upstream and Paul said that it's fine: Is the pyopenssl thread-safe? pyca/pyopenssl#424. But he made a remark about subinterpreters /
mod_wsgiand I know that CherryPy has some code that is probably not being tested in CI related tomod_wsgi. But this was nearly a decade ago. - https://stackoverflow.com/a/61826374/595220 hints that prior to OpenSSL 1.1.0 there would be some problems but not anymore. That version hit EOL 6 years ago: https://endoflife.date/openssl. So we shouldn't need to care about it.
- Is SSL_SESSION supposed to be thread-safe? openssl/openssl#25343 / https://crypto.stackexchange.com/a/41344/70030 / https://docs.openssl.org/1.0.2/man3/threads/#description claim that low-level
SSL_SESSIONisn't thread-safe. But given the PyOpenSSL issue, I'm inclined to believe that it's fine in the Python land. - I did some Git paleontology and the commit that added locking on Sep 12, 2006 cherrypy/cherrypy@f2e3c1c (f2e3c1c); being terrified of
exec(), I rewrote it to a metaclass on Sep 2, 2018 in e9e070b
That commit does not really explain anything but referencespyOpenSSL/tsafe.py. This was pre-PyPI which explains why things were being copied across projects and/or vendored. The first PyOpenSSL release on PyPI was 0.6 on Jun 4, 2008. So it's already two years past the CherryPy commit. We can't say for sure which version this was copied from. The very first import of PyOpenSSL from some other VCS into Git was on Feb 2, 2008 (pyca/pyopenssl@897bc25) but it's tagged as v0.14a1 while is probably an error or some weird historic artifact. However, that commit does contain theexec()snippet with locking that ended up in CherryPy — https://github.com/pyca/pyopenssl/blob/897bc2556fed43b76f6d1b14470c3e806df15af8/tsafe.py. On Jul 28, 2010, that module moved within the repo to https://github.com/pyca/pyopenssl/blob/94cc894795226537daf2f4e79e2d142ff51eacec/OpenSSL/tsafe.py (pyca/pyopenssl@94cc894#diff-eb4dc4606a8f26382d033c9859d6b5be8043e8bc511522f2e014e181bd09c09d). More history of this module: https://github.com/pyca/pyopenssl/commits/e41098f271dc96d6c411172bb11bbe5816f78710/src/OpenSSL/tsafe.py. It was deprecated on Sep 4, 2017 in PyOpenSSL v17.3.0 (Deprecate OpenSSL.tsafe pyca/pyopenssl#655 / Fixes #655 -- deprecate OpenSSL.tsafe pyca/pyopenssl#673) and removed fully on Nov 27, 2020 in v20.0.0 (Remove deprecated tsafe module. pyca/pyopenssl#913). The maintainers thought nobody uses it and weren't sure it was functional. At the time of deletion, it was still using the originalexec()(with an additional method added to the list throughout its life).
So I'd get rid for the hacks and made it as simple as possible, instead of trying to preserve them w/o a good recorded/explainable justification.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very thorough! But obviously removing the thread-safety is a potentially breaking change if anyone is relying on it.
d2672c9 to
85503d9
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #787 +/- ##
==========================================
- Coverage 79.60% 79.39% -0.21%
==========================================
Files 29 29
Lines 4261 4237 -24
Branches 542 539 -3
==========================================
- Hits 3392 3364 -28
- Misses 726 729 +3
- Partials 143 144 +1 |
85503d9 to
cdcab0c
Compare
4cda563 to
57ab3fb
Compare
|
@julianz- linking the thread where I posted more thoughts in case it gets lost in the UI: #787 (comment). |
0e19fce to
4e5decd
Compare
4e5decd to
2aa8da3
Compare
cheroot.ssl.pyopenssl removing SSLConnectionProxyMeta
| """Proxy attribute access to the underlying ``SSL.Connection``.""" | ||
| return getattr(self._ssl_conn, name) | ||
|
|
||
| def shutdown(self, how=None): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we should have something for sock_shutdown().
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unlike makefile() which I just had to add, the linter is not complaining about this method being missing. We could add for socket compatibility I guess, but it's just going to call the shutdown method. If we try to replicate the whole socket interface then wouldn't we need to add a bunch of other methods? Then we end up adding all kinds of cruft?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm mostly curious about the calling code using it rather than us reimplementing it. This class will get the implementation through inheritance.
@julianz- I just remembered that the calling code actually invokes sock_shutdown() and not shutdown(). See
Line 1543 in 22bb430
| 'sock_shutdown', |
I wonder if we could have interface that would be uniform across adapters and wouldn't have that ugly getattr() hack, which is effectively an abstraction leak.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe then we assume all connection adapters (including SSLConnection) implement sock_shutdown?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@julianz- yes, that's sort of what I was considering. But looking at
cheroot/cheroot/ssl/builtin.py
Lines 313 to 317 in 22bb430
| s = self.context.wrap_socket( | |
| sock, | |
| do_handshake_on_connect=True, | |
| server_side=True, | |
| ) |
socket.socket().
This means a separate PR as it feels like a significant effort to figure out. If you're up for it, please try composing a patch scoped to implementing this compatibility interface. It would probably need to come before this PR, though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok will give it a try
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm actually not sure if this constitutes a breaking change. SemVer talks about breaking changes in the context of API compatibility and the API seems to have remained intact. WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the API offered thread-safety and now suddenly we are saying these methods are not thread-safe that sounds like a breaking change to me, no? It's somewhat moot I guess if, in practice, no-one needs these methods to be thread-safe but just in case it's better to clear this is a potentially breaking change. If not a breaking change what category would be best?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Honestly, I'm torn. We could keep it breaking. Otherwise, it'd probably be misc or something like that, which is not very prominent in the change log.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure. That makes sense. I'm fine to move to misc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@julianz- I'm not opposed to keeping it breaking. Just thinking out loud. If it's breaking, I'd bump the major version component on release. Otherwise, it might be a minor change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is Breaking Bad? haha. i'm torn also. i changed to misc but if you want to go back to breaking let me know.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's keep it misc, I suppose.
4861bf3 to
4757cd2
Compare
cheroot/ssl/pyopenssl.py
Outdated
| :py:meth:`.shutdown()` is available for interface compatibility. | ||
| """ | ||
|
|
||
| def makefile(self, *args, **kwargs): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need this? Can we replicate the method signature exactly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
makefile() I added because the linter complained about makefile being abstract and needing an implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Which linter? Do you have logs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@julianz- I'm pretty sure the abstract method is in the adapter and not in SSL.Connection. Are you sure this is needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess it's not really abstract in SSL.Connection - it's just marked as not implemented. The linter interprets that as abstract?
cheroot.ssl.pyopenssl removing SSLConnectionProxyMetacheroot.ssl.pyopenssl.SSLConnection method
f0d0d53 to
b5cd936
Compare
| 'sock_shutdown', | ||
| self.socket.shutdown, | ||
| ) | ||
| shutdown = self.socket.sock_shutdown |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In https://github.com/cherrypy/cheroot/pull/787/files#r2498797283 I explained that this would require more work. Specifically because stdlib's socket.socket does not have this method. Also, this wouldn't need to go into a variable but called directly.
Removes custom thread-safety locking logic as the wrapper is not required to be thread-safe. The wrapper now inherits simply from SSL.Connection.
b5cd936 to
b19d1f4
Compare
Removes metaclass by using
__getattr__(self, name)to interceptmethod calls.